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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [3 Claim(s) 1-3,5-20 and 22-25 is/are pending in the application. 

4a) Of the above claim(s) 4, 21 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) IEI Claim(s) 1-3.5-14.17-20 and 22-25 is/are rejected. 

7) £3 Claim(s) 15. 16 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) E3 The drawing(s) filed on 27 October 0203 is/are: a)S accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
Priority under 35 U.S.C. §§119 and 120 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) £3 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) D The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 



Attachment(s) 

1) S Notice of References Cited (PTO-892) 

2) EH Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) 



4) □ Interview Summary (PTO-413) Paper No(s). . 

5) □ Notice of Informal Patent Application (PTO-152) 

6) □ Other: 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 11-03) 
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DETAILED ACTION 
Drawings 

1. The drawings were received on 27 October 1003. These drawings are accepted. 
Formal drawings should be submitted. 

Claim Rejections - 35 USC § 102 

2. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

3. Claims 9, 10, 13 and 17 are rejected under 35 U.S.C. 102(e) as being anticipated 
by Iwamura 6,501,515 B1. 

In regard to claims 9, 10, 13 and 17, see paper 4, 102(e) rejection of 
claims 9, 10, 13 and 17. It is noted that applicant indicated on page 11, first 
paragraph, of the response that claim 9 is a dependent claim. However, claim 9 
is actually an independent claim. 

Allowable Subject Matter 

4. The indicated allowability of claims 4, 6, 7, 1 1 , 12, 14, 21, 23 and 24 is withdrawn 
in view of the newly discovered reference(s) to Funayama et al. 6,332,038 B1. 
Rejections based on the newly cited reference(s) follow. 

Claim Rejections - 35 USC § 103 

5. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

6. Claims 1-3, 8, 9, 18-20 and 25 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Qiao 6,075,895 in view of Funayama et al. 6,332,038 B1. 
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In regard to claims 1 and 18, Qiao discloses an invention similar to that 
which is claimed in claims 1 and 18. See paper 4, 102 rejection of claims 1 and 
18 for similarities. Qiao does not disclose that determining the object in the 
image comprises the steps of obtaining a normalized color representation for a 
plurality of colors in each image, determining from training images an estimate of 
a probability distribution of normalized color values for an object class, and 
determining, for each pixel, a likelihood according to an estimated probability 
density of normalized color values for the object class. 

Funayama et al. discloses an image processing device that detects an 
object in an image. See column 2, lines 15-18, disclosing, "For example, when 
the foregoing specified object domain is a human face, by using a probability 
density function derived from a color distribution of human faces, skin areas of 
faces can be separated from the base image." Thus, Funayama et al. 
determines from training images an estimate of a probability distribution of color 
values for an object class and determines a likelihood according to an estimated 
probability density of color values for the object class. See column 14, lines 57- 
59, disclosing that "the separability measurement process was carried out 
for... all pixels of the base image"; thus, the likelihood is determined for each 
pixel. See column 13, line 44, disclosing that "separability is a normalized 
quantity". It is understood that the color values are normalized, as normalization 
is conventional and necessary in order to compare images. 
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It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Qiao by determining an object on 
the image by obtaining a normalized color representation for a plurality of colors 
in each image, determining from training images an estimate of a probability 
distribution of normalized color values for an object class, and determining, for 
each pixel, a likelihood according to an estimated probability density of 
normalized color values for the object class. One would have been motivated to 
make such a modification based on the teaching of Funayama et al. to use a 
probability density function derived from a color distribution of the object class in 
order to separate an object from an image, thereby determining an object in the 
image. Also, the normalization of color values is conventional and necessary in 
order to compare images for any purpose, including object determination. 

7. In regard to claims 2, 3, 8, 9, 19, 20 and 25, see paper 4, 102 rejection of claims 
2, 3, 8,9, 19,20 and 25. 

8. Claims 5 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Qiao 6,075,895 in view of Funayama et al. 6,332,038 B1 in further view of 
Iwamura 6,501,515 B1. 

In regard to claims 5 and 22, see paper 4, 103 rejection of claims 5 and 

22. 

9. Claims 6, 7, 23 and 24 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Qiao 6,075,895 in view of Funayama et al. 6,332,038 B1 as 
applied to claims 1 and 18 above, and further in view of Prasad et al. 5,680,481. 
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In regard to claims 6 and 23, Qiao in view of Funayama et al. discloses an 
invention similar to that which is claimed in claim 6. See rejection of claims 1 and 
18 for similarities. Qiao in view of Funayama et al. does not disclose that 
determining the trajectory comprises the steps of: determining a difference in a 
size of the object over a predetermined time period; determining a plurality of 
angles between a plurality of lines connecting successive centroids over the time 
period; and determining a feature vector according to the angles and lines. 

Prasad et al. Discloses in column 5, lines 60-64, "The first normalization 
step provided a set of facial measurements characteristic of the size of each 
speaker's face by averaging the vectors corresponding to all rested position 
times." Thus, Prasad et al. determines a difference in a size of the object over a 
pre-determined time period. 

Prasad et al. further discloses determining a plurality of angles between a 
plurality of lines connecting successive centroids over the time period. See 
figure 4. Also see column 8, lines 60-65, describing lines and angles connecting 
eye centroids and column 9, lines 28-31 disclosing lines and angles associated 
with a mouth area centroid. Further see column 4, lines 60-62, disclosing, "The 
centroids of each marker are tagged and linked from frame-to-frame to form a 
trajectory." 

Prasad et al. further discloses that the angles and lines connecting 
centroids determine a feature vector. The angles and lines are used to form a 
region of interest (ROI) coordinate system (see column 9, lines 36-41) and from 
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this, a feature vector is formed, (see figures 6-8 and column 10, lines 16-55, 
disclosing how the centroid connecting lines and angles are mapped to 
coordinate systems to extract peaks and valleys used to make up a visual feature 
vector." See column 10, lines 61-63, disclosing, "additional feature vector 
elements were created from the frame-to-frame (image-to-image) observations". 
Thus, the feature vector elements were formed from centroid-connecting lines 
and angles over time. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Qiao in view of Funayama et al. by 
determining a trajectory with the steps of: determining a difference in a size of the 
object over a predetermined time period; determining a plurality of angles 
between a plurality of lines connecting successive centroids over the time period; 
and determining a feature vector according to the angles and lines. One would 
have been motivated to make such a change based on the teachings of Prasad 
et al. to form a feature vector by calculating angles and lines connecting an 
object's centroids over time, and in order to form a feature vector "suitable for 
processing by a... classifier" from a data set. 
10. In regard to claims 7 and 24, Qiao in view of Funayama et al. further in view of 
Prasad et al. discloses an invention similar to that which is disclosed in claims 7 
and 24. See rejection of claims 1 , 18, 6 and 23 for similarities. Qiao in view of 
Funayama et al. does not disclose that classifying the trajectory further 
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comprises the step of classifying the feature vector according to a time-delay 
neural network, wherein a feature is of a fixed length. 

See Prasad et al., column 10, line 60 to column 1 1 , line 16, describing the 
elements of the visual feature vector and disclosing that the "resulting visual 
feature vector includes 22 elements which are used as the visual data vector 
input to a TDNN [time delay neural network] speech classifier as shown in FIG. 
10." Note elements 3), 5), and 6) of the feature vector, which involve calculations 
based on a change in time. Thus, a feature is of a fixed length. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Qiao in view of Funayama et al. by 
classifying the feature vector according to a time-delay neural network, wherein a 
feature is of a fixed length. One would have been motivated to make such a 
change based on the teaching of Prasad et al. that, "In order to accommodate 
utterances that may be of variable length, as well as somewhat unpredictable in 
the time of utterance onset, the neural network employed... was chosen to be a 
time delay neural network (TDNN)". Because of this teaching, one would have 
been motivated to use a time delay neural network in any type of motion tracking, 
as motion of objects "may be of variable length, as well as somewhat 
unpredictable in the time of... onset". 
11. Claims 11 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Iwamura 6,501,515 B1 in view of Jeannin 6,587,574 B1. 
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In regard to claim 1 1 , Iwamura discloses an invention similar to that which 
is claimed in claim 1 1 . See paper 4, 102 rejection of claim 9 for similarities. 
Iwamura does not disclose that the reference point (hand) is characterized by 
size and location of a centroid of the reference point (hand) in each image. 

Jeannin discloses in column 8, lines 30-34 that, "Each object's centroid 
provides a single reference point within each frame that is used to analyze the 
object's movement. The trajectory of the object's centroid is the basis for 
describing the motion or trajectory of the object from frame 201 to frame 204 [see 
figure 2]." Thus, the reference point is characterized by location of a centroid of 
an object in each image. Jeannin further discloses in column 12, lines 57-59, 
that "z information may be deduced from the size variation of the object between 
consecutive frames." Thus, a reference size is inherent, and an object's 
reference point is characterized by size of the object. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Iwamura by having the reference 
point characterized by hand size and a location of a centroid of the hand in each 
image. One would have been motivated to make such a change based on the 
teaching of Jeannin to use an object's centroid as a single reference point to 
analyze the object's movement within each frame and to use change in the 
object's size to determine z information size. 
12. In regard to claim 12, Iwamura discloses an invention similar to that which is 
disclosed in claim 12. See paper 4 102 rejection of claim 9 for similarities. 
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Iwamura does not disclose that the first translation is one of a forward and a 
backward translation, wherein the first translation is characterized by a large 
change in hand size and a relatively small change in a centroid of the hand. 

Jeannin discloses in column 12, lines 52-54, that, "The speed or velocity 
of the object for each time instant may be calculated as the local first order 
derivative of the centroid position, normalized by the image size." Also, see 103 
rejection of claim 1 1 disclosing that Jeannin uses object size and centroid 
location to classify trajectory information, in the z direction in particular. Thus, 
any forward and backward translation (z direction translation) is characterized by 
object size and centroid location. It is inherent that for z direction translation, this 
would be a large change in hand size and a relatively small change in centroid 
location of the object. This could be the first or any translation in the object's 
trajectory. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Iwamura by having the first 
translation be one of a forward and a backward translation, wherein the first 
translation is characterized by a large change in hand size and a relatively small 
change in a centroid of the hand. One would have been motivated to make such 
a change based on the teaching of Jeannin to use the location of an object's 
centroid as a reference and to deduce z information from an object's size. Thus, 
the first or any translation in a z (forward or backward) direction could be 
identified. 
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13. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Iwamura 
6,501 ,51 5 B1 in view of Prasad et al. 5,680,481 . 

In regard to claim 14, Iwamura discloses an invention similar to that which 
is disclosed in claim13. See paper 4, 102 rejection of claim 9 for similarities. 
Iwamura does not disclose that detecting the second translation further 
comprises the step of determining a normalized vector between two centroids 
c.sub.t and c.sub.t-1 as a feature vector, wherein there are three output patterns. 

Prasad et al. discloses determining a plurality of angles between a 
plurality of lines connecting successive centroids over the time period. See 
figure 4. Also see column 8, lines 60-65, describing lines and angles connecting 
eye centroids and column 9, lines 28-31 disclosing lines and angles associated 
with a mouth area centroid. Further see column 4, lines 60-62, disclosing, "The 
centroids of each marker are tagged and linked from frame-to-frame to form a 
trajectory." Prasad et al. discloses that the angles and lines connecting centroids 
determine a feature vector. The angles and lines are used to form a region of 
interest (ROI) coordinate system (see column 9, lines 36-41) and from this, a 
feature vector is formed, (see figures 6-8 and column 10, lines 16-55, disclosing 
how the centroid connecting lines and angles are mapped to coordinate systems 
to extract peaks and valleys used to make up a visual feature vector." Thus a 
normalized vector is formed between centroids. See column 10, lines 61-63, 
disclosing, "additional feature vector elements were created from the frame-to- 
frame (image-to-image) observations". Thus, the feature vector elements were 
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formed from centroid-connecting lines and angles over time, and the normalized 
vector is formed between two centroids in time. As the feature vector is mapped 
to a TDNN (see rejection of claims 7 and 24), for speech recognition, it is 
inherent that there are at least three output patterns. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to modify the invention of Iwamura by detecting the second, 
or any, translation by determining a normalized vector between two centroids 
c.sub.t and c.sub.t-1 as a feature vector, wherein there are three output patterns. 
One would have been motivated to make such a modification based on the 
teaching of Prasad et al. to form a feature vector from centroids throughout time 
to recognize speech and because such a tracking of trajectory could be used to 
recognize any form of motion. 



14. Claims 15 and 16 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 



15. The prior art made of record and not relied upon is considered pertinent to 
applicants disclosure. 

Smith et al. 6,128,003 discloses a hand gesture recognition system and 
method which includes calculation of rotational vectors using hand centroid 
locations. 



Allowable Subject Matter 



Conclusion 
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Stork et al. 5,771 ,306 discloses a method for speech recognition using 
centroid tracking. 

Dakss et al. 6,642,940 B1 discloses a method for object identification that 

uses probability density distributions of pixel color values. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Laurel E LeFlore whose telephone number is (703) 305- 
8627. The examiner can normally be reached on Monday-Friday 8-4:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph Mancuso can be reached on (703) 305-3885. The fax phone 
number for the organization where this application or proceeding is assigned is (703) 
872-9306. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. /7 



